Fingerprint database maintenance method and system

ABSTRACT

A method of maintaining a database comprising a fingerprint of and an associated set of metadata for each of a number of multimedia objects. Respective portions ( 201, 202, 203, 204, 205 ) of the database are distributed over respective file sharing clients ( 101 - 105 ) connected to a file sharing network ( 100 ) arranged for sharing said number of multimedia objects. File sharing clients ( 101 - 105 ) can maintain their own respective portions ( 201 - 205 ) of the database, or transmit fingerprints and metadata to another file sharing client. In the latter case, the other file sharing client is preferably a supermode in the file sharing network ( 100 ).

The invention relates to a method of maintaining a database comprising afingerprint of and an associated set of metadata for each of a number ofmultimedia objects. The invention further relates to a file sharingclient, a computer program product and a file sharing network.

Fingerprints of human beings are already used for over a hundred yearsto identify people. Conceptually a fingerprint can be seen as a shortsummary, which is unique for every single human being. Recently agrowing interest is seen in the field of multimedia processing tocompute fingerprints of multimedia objects. In order to qualify twomultimedia objects as the same, instead of comparing the multimediaobjects itself, only their fingerprints are compared. A fingerprint of amultimedia object is a representation of the most relevant perceptualfeatures of the object in question. Such fingerprints are sometimes alsoknown as “(robust) hashes”.

In most systems using fingerprinting technology, the fingerprints of alarge number of multimedia objects along with their associatedrespective metadata are stored in a database. The term “metadata” refersto information such as the title, artist, genre and so on for amultimedia object. The metadata of a multimedia object is retrieved bycomputing its fingerprint and performing a lookup or query in thedatabase using the computed fingerprint as a lookup key or queryparameter. The lookup then returns the metadata associated with thefingerprint.

There are several advantages in storing fingerprints for multimediaobjects in a database instead of the multimedia content itself. To namea few:

-   1. The memory/storage requirements for the database are reduced.-   2. The comparison of fingerprints is more efficient than the    comparison of the multimedia objects themselves, as fingerprints are    substantially shorter than the objects.-   3. Searching in a database for a matching fingerprint is more    efficient than searching for a complete multimedia object, since it    involves matching shorter items.-   4. Searching for a matching fingerprint is more likely to be    successful, as small changes to a multimedia object (such as    encoding in a different format or changing the bit rate) does not    affect the fingerprint.

An example of a method of generating a fingerprint for a multimediaobject is described in International patent application WO 02/065782(attorney docket PHNL010110), as well as in Jaap Haitsma, Ton Kalker andJob Oostveen, “Robust Audio Hashing For Content Identification”,International Workshop on Content-Based Multimedia Indexing, Brescia,September 2001.

In large-scale systems, the fingerprint database has to be distributedover a considerable number of fingerprint servers to be able to handleall the search requests and to store all the fingerprints. Furthermore,the database has to be kept up to date. For example, in the case ofaudio fingerprinting, the fingerprints of newly released songs have tobe added. Both the necessary servers and keeping the database up-to-datemake the system very costly.

It is an object of the invention to provide a method according to thepreamble, which is cheaper than the known method.

This object is achieved according to the invention in a methodcomprising distributing respective portions of the database overrespective file sharing clients connected to a file sharing networkarranged for sharing said number of multimedia objects.

Using this method, it is no longer necessary to actively go out and buycontent e.g. on CD, or to find out the metadata for content yourself. Byexploiting the objects and the metadata available from the file sharingclients on the network, fingerprints and metadata can be collected in avery cheap and efficient way. These clients already make the objectsavailable for anyone to download, so buying these objects becomesunnecessary. Further, typically the objects are made available togetherwith metadata, so this metadata can be used as well.

Distributing the database over the file sharing network has theadditional advantage that no dedicated database servers or managementsystems are necessary. The file sharing network already contains apotentially large number of interconnected computers, which provide awell-suited basis for maintaining such a database.

Also, the method according to the invention is more scalable than priorart methods. When more users join the file sharing networks, the numberof requests for metadata will increase, and so the requirements on thedatabase server(s) must be increased if a satisfactory response time isdesired. However, when the database is distributed over the clients inthe file sharing network, then more computers will become available onthe network when new users join the network. The extra computing power,storage and connectivity provided by these new computers can then beused to maintain a portion of the database. This way, the capabilitiesof the distributed database scales together with the demand.

It is a further object of the invention to provide a file sharing clientcomprising a storage for storing one or more multimedia objects, sharingmeans for sharing a multimedia object in the storage with other filesharing clients on a file sharing network, fingerprinting means forcomputing a fingerprint and obtaining a set of metadata for themultimedia object shared by the sharing means, and for adding thecomputed fingerprint and obtained set of metadata to a databasedistributed over the file sharing clients connected to the file sharingnetwork.

Such a file sharing client is capable of participating in the method ofmaintaining a database as outlined above. Because the fingerprinting anddata collecting means are integrated in the file sharing client, userswho install the file sharing client also automatically install thenecessary means to help maintain the distributed database. Thus, whenthey subsequently join the file sharing network, their computing power,connectivity and storage becomes available to the network, and extendsthe capabilities of the distributed database.

In an embodiment the file sharing client further comprises DBMS meansfor maintaining a portion of the distributed database. By installingsuch database management system means in the file sharing client, anyonewho installs the client (usually on a computer system) also installs theDBMS means and so can contribute to the maintenance of the distributeddatabase.

In a further embodiment the size of the portion of the distributeddatabase maintained by the DBMS means is made dependent on theperformance of a computer system on which the client is running. Forexample, bandwidth restrictions, CPU speed and/or available workingmemory (RAM) could be taken into account. This way, a slow computerwould not be burdened with a large fingerprint database server.

In a further embodiment the DBMS means are arranged for adding thecomputed fingerprint and obtained set of metadata to the respectiveportion. This way, the distributed database is updated with newfingerprints and sets of metadata from multimedia objects that arepresent on the file sharing client. Each client now maintains a portionof the distributed database containing at least objects present in itsown storage.

In a further embodiment the fingerprinting means are arranged fortransmitting the computed fingerprint and the obtained set of metadatato another file sharing client on the file sharing network. This way,data to be stored in the database can be distributed via the filesharing network so that it can be stored in a portion managed by anarbitrary client arranged for managing that portion.

In a variant of the above embodiment the other file sharing client is asuper node in the file sharing network. Super nodes are clients whichhave sufficient bandwidth, processing power and memory. A normal clientconnects to the network by connecting to a super node and sends the listof the files to be shared to the super node. A super node hasconnections to a number of clients and furthermore is also connected toa number of other super nodes. Because of their larger capacities interms of memory, processing power and bandwidth, they are better suitedto manage a portion of the distributed database.

In a further embodiment the transmitting is done simultaneously withtransmitting a multimedia object to the other file sharing client. Thesefingerprints are relatively small (in the order of ten kilobytes, asopposed to several megabytes for a typical multimedia object) and sowill not affect the performance of the client. This provides a way todistribute the database with fingerprints and metadata in an arbitraryfashion over the clients on the network.

In a further embodiment the fingerprinting means are arranged forcomputing the fingerprint and obtaining the set of metadata for themultimedia object when the multimedia object is being stored in thestorage. By computing the fingerprint at this time, it is achieved thatmetadata for any newly obtained multimedia object is automatically addedto the distributed database.

It is a further object of the invention to provide a computer programproduct arranged for causing a general purpose computer to function asthe file sharing client according to the invention.

It is a further object of the invention to provide a file sharingnetwork comprising at least one file sharing client according to theinvention.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments shown in the drawing, inwhich:

FIG. 1 schematically shows a file sharing network comprising pluralclients; and

FIG. 2 schematically shows a file sharing client in more detail.

Throughout the figures, same reference numerals indicate similar orcorresponding features. Some of the features indicated in the drawingsare typically implemented in software, and as such represent softwareentities, such as software modules or objects.

FIG. 1 schematically shows a file sharing network 100 comprising pluralfile sharing clients 101, 102, 103, 104 and 105. Although shown here asa physical network, with direct connections between the clients 101-105,the network 100 is best regarded as a conceptual or virtual network.That is, it is not necessary that all clients 101-105 are physically ornetwork-wise directly connected to each other all the time. All that isneeded is that one client “on the network” can obtain files or objectsfrom another client. Also, even when direct client-to-client connectionsare used, it is not necessary that all clients are connected to allother clients.

The network 100 may comprise a server 110, which performs a directoryservice for the clients 101-105. To connect to the file sharing network100, a client 101 submits a list of the files (or objects) it wants toshare to the server 110. The server 110 combines the lists it receivesfrom all the clients connected to the network 100. Other clients 102-105can then connect to the server 110 and browse the combined list orsearch for specific objects on the list. They can subsequently contactthe client that has the object they are looking for, and obtain(download) it from that client directly. This way, the server 110 doesnot directly participate in the sharing of files or objects between theclients 101-105. This approach is well known in the worldwide Napsterfile sharing network.

It is also possible to realize the network 100 without the server 110.In that case, a client 101 connects to the network 100 by connecting toone or more other clients 102-105 that are already on the network 100. Aclient searches the network by sending a search request to the clientsit is connected to. These clients examine their list of objects whichthey share, and return a result if the requested object is in that list.Furthermore, the request is forwarded to other clients connected tothese clients. This way, the request is distributed throughout theentire network 100 until it is received by a client which can fulfillit, or until all clients have received it and none are able to fulfillit.

Such an embodiment is known from e.g. the Gnutella file sharing network.A disadvantage of this embodiment is that the network 100 is notscalable. Gnutella-like networks currently for example cannot support 1million clients. Furthermore the network becomes slow if there are anumber of “slow” computers, i.e. computers with limited bandwidth to thenetwork 100, processing power and/or memory.

Alternatively the client 101 can, after connecting to the one or moreother clients 102-105, submit its list of files or objects it wants toshare to those other clients 102-105. The list is then passed on to allthe clients on the network 100. This way, all clients know which clientshave which files or objects available, and can contact that clientdirectly.

The known KaZaa file sharing network also operates without a server 110,but to overcome the above-mentioned problem uses two types of clients: asuper node and a “normal” client. Super nodes are clients which havesufficient bandwidth, processing power and memory. A normal clientconnects to the network by connecting to a super node and sends the listof the files to be shared to the super node. A super node hasconnections to a number of clients and furthermore is also connected toa number of other super nodes.

A super node is at the same time also a normal client. That is, for theuser the fact that his computer is a super node is transparent. When auser wants to search for a file, his client sends a request to the supernode(s) to which his client is currently connected. The super nodesreturns the matching files, that are in the lists send by its clients.Furthermore the super node forwards the request, if necessary, to allthe super nodes to which it is connected in a fashion similar to the onedescribed above in the Gnutella embodiment. However, since theconnections between super nodes have a large bandwidth this approach ismuch faster than the Gnutella networks. Furthermore it can be scaled upto millions of clients.

Such file sharing networks, typically referred to as peer-to-peer or P2Pfile sharing networks, have an enormous popularity. Well known examplesof these networks are: Napster, Musiccity, Gnutella, Kazaa, Imesh andBearshare. Once users have installed the appropriate client software ontheir personal computers, they can share their files and they are ableto download files shared by other users. The clients 101-105 may beconnected to a network such as the Internet, which facilitates theestablishment of the file sharing network 100. A client could e.g. use adirect TCP/IP connection to another client to obtain a file or object.

On the most popular networks, usually over 500,000 people are connectedsimultaneously. At the time of writing, people are mostly sharing musicfiles (often in the MP3 format), but the sharing of movies is gainingpopularity. The term “multimedia object” will be used to denote filescontaining music, songs, movies, TV programs, pictures and other typesof binary data, but also textual data can be shared in this fashion. Itis to be noted that a multimedia object may be made up of severaldifferent files.

The network 100 also comprises a distributed database. The distributeddatabase is made up of several respective portions 201-205, each ofwhich is maintained by a respective one of the clients 101-105. Thiswill be explained below with reference to FIG. 2.

FIG. 2 shows the file sharing client 101 in more detail. The filesharing client 101 is preferably realized as a personal computer onwhich file sharing software 301 is running, as is well-known in the art.The file sharing software 301 typically makes use of a networking module302, such as the TCP/IP stack available in modern operating systems. Astorage 303 contains one or more multimedia objects which are shared bythe file sharing software 301. Such a storage 303 would typically be adirectory on a hard disk. In some cases, the storage 303 may contain aseparate portion in which downloaded multimedia objects are stored. Thisportion, typically also a directory, is not necessarily the same as thedirectory in which multimedia objects to be shared are stored.

The file sharing client 101 also comprises a fingerprinting module 304,which can compute a fingerprint from a multimedia object. As mentionedabove, one method for computing a fingerprint is described inInternational patent application WO 02/065782 (attorney docketPHNL010110), although of course any method for computing a fingerprintcan be used. The fingerprinting module 304 also obtains a set ofmetadata for the multimedia object. Often this set of metadata isincluded in or with the multimedia object, so that obtaining the set ofmetadata is done automatically when obtaining the multimedia object.

The fingerprinting module 304 is preferably realized as one or morehardware or software modules, for example as a plug-in module in thefile sharing software 301 running on the client 101.

The fingerprinting module 304 can compute the fingerprints frommultimedia objects in the storage 303. The set of metadata for themultimedia object can similarly be obtained by simply reading it fromthe multimedia object on the storage 303. For instance, a multimediaobject with music in the popular MP3 format often contains metadata asan ID3 ‘tag’ at the end of the object.

As computing a fingerprint for multimedia object may be CPU-intensive,care must now be taken to avoid consuming too much CPU power. Doing somight upset the user of the file sharing software as he sees itinterfere with his normal use of the system.

The fingerprint can be computed upon user request or alternatively inthe background. In the latter case, it is preferred to periodically scanthe shared drives or directory for new multimedia objects for which nofingerprint has been computed yet. If any such objects are found, afingerprint is computed automatically. If no metadata is available forsuch an object, the user could be prompted to enter a set of metadata.

In any case, once the fingerprinting module 304 has computed afingerprint for a multimedia object, and has obtained a set of metadatafor the multimedia object, it includes fingerprint and set of metadatain the distributed database 201-205. Preferably, the fingerprint and theset of metadata are included in the portion 201 maintained by DBMSmodule 305.

A fingerprint Database Maintenance (DBMS) module 305 maintains theportion 201 of the distributed fingerprint database. The database 201contains fingerprints and associated sets of metadata. The database 201will typically contain for each shared multimedia object a fingerprintand one associated set of metadata, unless of course the storage 303contains multiple copies of one particular multimedia object.

Additionally, the database 201 could be extended with fingerprints andmetadata for multimedia objects downloaded by the file sharing client101 from other file sharing clients 102-105 on the network 100. Afingerprint for a multimedia object can be computed while that object isbeing downloaded. Some methods of computing a fingerprint operate onsmall portions of a multimedia object at a time. For example, theabove-mentioned European patent application computes a “sub-fingerprint”for every three seconds of audio data in the multimedia object, andconstructs the actual fingerprint from all the sub-fingerprints.Computing the sub-fingerprints can then start once three seconds worthof data has been received.

If the metadata for that object is available as well, the fingerprintand metadata can be included in the database 201 before the object isdownloaded completely. If during this process it is determined that thefingerprint is already in the database 201, it is very likely that theuser already has a copy of this particular multimedia object in hispossession. The user could then be warned, so that he can abort thedownloading.

When the file sharing client 101 is downloading a multimedia object fromanother client 102, the client 101 can also download one or morefingerprints with associated sets of metadata from the client 102. Thesefingerprints are relatively small (in the order of ten kilobytes, asopposed to several megabytes for a typical multimedia object) and sowill not affect the performance of the client 101. This provides a wayto distribute the database with fingerprints and metadata in anarbitrary fashion over the clients 101-105 in the network 100.

In the KaZaa file sharing network, the super nodes are preferably usedto distribute fingerprints and metadata over the network 100. In anetwork like the Napster file sharing network, it could be the centralserver that distributes the fingerprints.

Obtaining the right metadata can also be assisted by super nodes orcentral servers. A client submits a search request for a particularfingerprint to the super node to which it is connected. The super nodepasses on the request to the other super nodes. Without a central serverthat filters the sets of metadata in the database to determine adefinite set, the super node would probably receive multiple answers tothe query. The super node can then apply majority voting or anothertechnique to determine a definite set of metadata which is then suppliedback to the client that submitted the request.

For example, suppose that the sets of metadata received in response to asearch request for a particular fingerprint are as follows:

-   1. (artist=“Jewwel”, title=“Hands”)-   2. (artist=“Jewel”, title=“Hands”)-   3. (artist=“Jewel”, title=“Hnds”)-   4. (artist=“Jewel”, title=“Hands”)-   5. (artist=“Jewel”, title=“Hands”)

It can easily be seen that in this example four out of five sets givethe name of the artist as “Jewel”, while only one gives the name as“Jewwel”. Using the simple approach that the majority wins, the definiteset of metadata would give the name of the artist as “Jewel”. Similarly,four out of five sets give the title of the song as “Hands”, and so thedefinite set of metadata would also give the title of the song as“Hands”. The same approach can of course be used for other types ofmetadata included in the sets, such as album title, publication year,genre, URL for the artist's Website and so on.

Other, more advanced techniques for automatically determining a definitevalue from a plurality of candidate values can of course also be used.Such techniques are common in the field of intelligent agents, wherethey are used to eliminate noise from information received by an agent.They include decision tree pruning and cross validation. What exactlyconstitutes a “sufficient number” depends on the technique used.

It is observed that not all sets of metadata are necessarily complete.For example, one set of metadata might contain only the title and thename of the artist for a particular song, while another might alsocontain the title of the album from which the song was obtained and theyear of publication of the album. So the above process should beperformed on the individual types of metadata, e.g. once for the titlebased on all the available titles, once for the artist's name based onall the available artist names, once for the year of publication, and soon. This way, a definite set of metadata is obtained which is asextensive as possible, i.e. which includes not only title and artist butalso album title and publication year. Such an extensive definite set ofmetadata is the most valuable.

The super node could subsequently update its own database with thedefinite set, so as to avoid having to pass on the query again to allthe other super nodes every time one of his clients submits that queryagain. However, this runs the risk that his information will be outdatedat some time.

The size of the portion of the distributed database 201 maintained byDBMS module 305 could be made dependent on the performance of thepersonal computer on which it is running. For example, bandwidthrestrictions, CPU speed and/or available working memory (RAM) could betaken into account. This way, a slow computer would not be burdened witha large fingerprint database server.

The file sharing clients 101-105 can make at least a portion of thedatabase 201-205 available to others. This can be done e.g. by offeringa search interface through which clients can submit a fingerprint andreceive a set of metadata in return. Various methods of retrieving froma database a set of metadata associated with a submitted fingerprint areknown from the above-mentioned International patent application WO02/065782 (attorney docket PHNL010110), as well as from Internationalpatent application WO 02/058246 (attorney docket PHNL010532). Othermethods can of course also be used.

If a particular client 101 cannot find a set of metadata associated withthe submitted fingerprint in its portion 201 of the distributeddatabase, it could forward the submitted fingerprint to another client102 to which it is connected in the file sharing network 100. The otherclient 102 is preferably a super node in the file sharing network 100,if the network 100 comprises super nodes. The other client 102 couldsimilarly forward the submitted fingerprint if it cannot find such a setin its portion 202, and so on until one of the clients 101-105 findssuch a set of metadata in its portion 201-205, or until all clients101-105 in the file sharing network 100 have failed to find such a set.

The contents of the distributed database 201-205 can be made availablefor free, or only to paying subscribers. Alternatively, a fee could becharged for every query performed on the database. The amount ofmetadata returned to the client in response to submitting a fingerprintcould also be varied: the free service returns only artist and title,and the subscription-based service returns all the metadata available inthe database, for example.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim. The word “comprising” does notexclude the presence of elements or steps other than those listed in aclaim. The word “a” or “an” preceding an element does not exclude thepresence of a plurality of such elements.

The invention can be implemented by means of hardware comprising severaldistinct elements, and by means of a suitably programmed computer. Inthe device claim enumerating several means, several of these means canbe embodied by one and the same item of hardware. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage.

1. A method of maintaining a database comprising a fingerprint of and anassociated set of metadata for each of a number of multimedia objects,the method comprising distributing respective portions of the databaseover respective file sharing clients connected to a file sharing networkarranged for sharing said number of multimedia objects.
 2. A filesharing client comprising a storage for storing one or more multimediaobjects, sharing means for sharing a multimedia object in the storagewith other file sharing clients on a file sharing network,fingerprinting means for computing a fingerprint and obtaining a set ofmetadata for the multimedia object shared by the sharing means, and foradding the computed fingerprint and obtained set of metadata to adatabase distributed over the file sharing clients connected to the filesharing network.
 3. The file sharing client of claim 2, furthercomprising DBMS means for maintaining a portion of the distributeddatabase.
 4. The file sharing client of claim 3, in which the size ofthe portion of the distributed database maintained by the DBMS means ismade dependent on the performance of a computer system on which it isrunning.
 5. The file sharing client of claim 3, in which the DBMS meansare arranged for adding the computed fingerprint and obtained set ofmetadata to the respective portion.
 6. The file sharing client of claim2, the fingerprinting means being arranged for transmitting the computedfingerprint and the obtained set of metadata to another file sharingclient on the file sharing network.
 7. The file sharing client of claim6, in which the other file sharing client is a super node in the filesharing network.
 8. The file sharing client of claim 6, in which thetransmitting is done simultaneously with transmitting a multimediaobject to the other file sharing client.
 9. The file sharing client ofclaim 2, the fingerprinting means being arranged for computing thefingerprint and obtaining the set of metadata for the multimedia objectwhen the multimedia object is being stored in the storage.
 10. Acomputer program product arranged for causing a general purpose computerto function as the file sharing client of claim
 2. 11. A file sharingnetwork comprising at least one client as claimed in claim 2.